DA - GTM Jong

The Generative Topographic Mapping (GTM), also known as a principled alternative to the SelfOrganizing Map (SOM), has been developed for modeling the probability density of data and its visualization in a lower dimension. Contrast to the SOM which does not define a density model [1], the GTM defines explicit probability density of data and aims to find an optimized model by using the Expectation-Maximization (EM) algorithm. Although the EM algorithm [3] has been widely used to solve optimization problems in many machine learning algorithms, such as the K-Means for clustering, the EM has a severe limitation, known as the initial value problem, in which solutions can vary depending on the initial parameter setting. To overcome such a problem, we have applied the Deterministic Annealing (DA) algorithm to GTM to find more robust answers against random initial values. The core of DA algorithm is to find an optimal solution in a deterministic way, which contrast to a stochastic way in the simulated annealing [8], by controlling the level of randomness. This process, adapted from physical annealing process, is known as a cooling schedule in that an optimal solution is gradually revealed by lowering randomness. At each level of randomness, the DA algorithm chooses an optimal solution by using the principle of maximum entropy [6, 5, 7], a rational approaches to choose the most unbiased and non-committal answers for given conditions. The DA algorithm [10] has been successfully applied to solve many optimization problems in various machine learning algorithms and applied in many problems, such as clustering [4, 10] and visualization [9]. Ueda and Nakano has developed a general solution of using DA to solve the EM algorithms [11]. However, not many researches have been conducted to research details of processing the DA algorithm. In this paper, we will tackle down more practical aspects in using DA with GTM. The main contributions of our paper are as follow: